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Abstract 

Reporting and interpretation of effect sizes and structure coefficients in multiple regression 
results is important for good practice. The purpose of this paper is to investigate the use and 
interpretation of effect size (ES) and structure coefficients in multiple regression analyses in two 
mathematics and science education journals. Published studies in the Journal for Research in 
Mathematics Education and School Science and Mathematics were inspected. Results suggest 
that effects are generally reported by researchers using multiple regression, but structure 
coefficients are seldom examined to help evaluate variable importance. 
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Use and Interpretation of Effect Sizes and Structure Coefficients 
in Mathematics and Science Education Journals 

Multiple regression analysis is a common tool used in social science research. Multiple 
regression can be useful in both predictive and explanatory research (Pedhazur, 1997). More 
than three decades ago, Jacob Cohen (1968) demonstrated that multiple regression was the 
univariate general linear model (GLM) and subsumed other univariate methods as special cases. 
Knapp (1978) later demonstrated canonical correlation as the multivariate GLM (see also 
Bagozzi, Fomell & Larcker, 1981; Henson, 2000; Thompson 1991). 

Because all GLM analyses are correlational in nature, they all yield r-type effect sizes 
that should be both reported and interpreted (cf. Cohen, 1994; Henson & Smith, 2000; 
Thompson, 1996). They also invoke weights (e.g., beta weights) that are applied to observed 
variables to create synthetic variables (e.g. Yhat predicted scores) that become the focus of the 
analysis. Therefore, the relationship between the observed variables and the synthetic variables 
becomes important in determining the value of the observed variable. The correlation between 
an observed and synthetic variable is called a structure coefficient, which are also present in all 
GLM analyses (Courville & Thompson, 2001; Thompson & Borrello, 1985). In regression, for 
example, the correlation (structure coefficient) between a predictor and the Yhat predicted scores 
informs the researcher as to the potential contributory value of the predictor to the effect 
observed. 

Further, within any GLM analysis such as multiple regression, researchers are typically 
faced with a two-stage process of result interpretation. Stage one concerns whether the 
researcher has a result that is noteworthy, which historically has been evaluated with statistical 
significance testing. More recently, the role and import of effect sizes have seen increased use, 
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including a mandate to “always” report effects because such reporting is “essential” to good 
research practice (Wilkinson & APA Task Force on Statistical Inference, 1999, p. 599). 

Once a noteworthy effect is observed, then (and only then), the researcher likely is 
concerned with determining what variables contributed to the effect. This is typically done by 
looking at standardized weights, but should also include examination of structure coefficients. 
For example, in multiple regression, a noteworthy R 2 effect is generally followed by study of the 
predictor beta weights (or unstandardized b weights) but should also invoke examination of the 
predictor structure coefficients. Courville and Thompson (2001) provide a thorough discussion 
on the definition and role of structure coefficients in multiple regression. Henson (2002) 
discusses structure coefficients in multivariate analyses. 

Purpose 

The purpose of the present paper is to investigate if suggested reporting standards of 
statistical information are being followed in two mathematics and science education journals. 
Focus was on the use and interpretation of effect size and structure coefficients in multiple 
regression applications. The literature suggests effect size reporting is generally sparse 
(DeVaney, 2001; Henson & Smith, 2000; Thompson, 2001), and the interpretation of effect sizes 
occurs even less often. However, Kirk (1996) observed higher rates of reporting for journals that 
frequently published articles employing multiple regression. Kirk postulated that effect reporting 
was greater for these journals because statistical software packages routinely provide R 2 effects 
in their outputs. This begs the question of whether researchers are thoughtfully including effects 
or if reporting is a function of what the computer output provides. Although an effect is 
reported, whether the effect is interpreted is another matter altogether. 
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Further, the reporting and interpretation of structure coefficients tends to be the 
exception rather than the rule. For the present study, I expected to observe moderate frequency 
of effect size reporting and very minimal use of structure coefficients. 

Statistical Significance Testing and Effect Size 

Statistical significant testing became common practice in the early to mid-1900’s with its 
introduction into textbooks (DeVaney, 2001). Fan (2001) stated “..., statistical significance 
testing answers the question: What is the probability of obtaining an observed sample statistic 
. . . when the population has a known parameter value . . .?” (p. 275). Statistical significance 
testing is studied in most statistics courses, which often leaves graduate students with the idea 
that statistical significance testing is the most important part of the statistical interpretation. 
Further, the common g-value is routinely the only statistic interpreted in educational research 
(Plucker, 1 997). 

Statistical significance testing has supporters and critics. The supporters take the position 
that when used correctly, statistical significance testing describes valuable information 
(DeVaney, 2001; Henson & Smith, 2000). Critics express that statistical significance testing is 
often misinterpreted as practical significance (DeVaney, 2001; Henson & Smith, 2000). The 
dependence of statistical significance testing on sample size often confuses correct 
interpretations of statistical significance (DeVaney, 2001; Fan, 2001; Plucker, 1997). With the 
apparent limitations of statistical significance tests, effect sizes have been advocated for result 
interpretation (Fan, 2001). 

Effect size in multiple regression typically refers to a variance-accounted-for statistic 
known as R 2 . Effect size does not replace statistical significance testing, and effect size 
reporting should complement statistical significance testing. After reporting the results of 
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significance testing, effect sizes should be reported and interpreted. Plucker (1997) reported that 
researchers in the social sciences are encouraged to provide effect size or the information for the 
calculation of the effect size in the results of studies. The fifth edition of the APA Publication 
Manual expects the reporting of effect size (APA, 2001). However, as Henson and Smith (2000) 
noted, “A more appropriate measure of effect size use would include an assessment of whether 
researchers both report and interpret their obtained effect sizes” (p. 290, emphasis added). 

Henson and Smith (2000) suggested that the trend toward reporting of effect size is 
progressing slowly. Even with the encouragement to report effect size by fourth edition of the 
APA Manuel (APA, 1994), the reporting of effect size is often neglected (DeVaney, 2001). 
Eleven empirical studies have been completed that show this encouragement by the APA has 
been ineffective (Thompson, 2001). 

Therefore, one purpose of this research seeks to determine the extent to which authors of 
mathematics and science education studies report and interpret effect size. 

Beta Weights and Structure Coefficients 

After finding noteworthy effects, the researcher is charged with determining the variables 
that contributed to those effects. In regression and other GLM analyses, this determination 
should invoke study of both standardized weights and structure coefficients. 

Beta weights are standardized coefficients (i.e., regression coefficients generated from 
standardized data) that suggest the independent variable’s predictive contribution to the 
dependent variable. Beta weights are easy to determine as they are generated in the output of 
multiple regression computer programs. However, beta weights are affected by collinearity, 
when predictor variables are correlated with each other. Hence, beta weights may yield distorted 
interpretations of variable import (Thompson & Borrello, 1985). 
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Structure coefficients in multiple regression are the correlations between the predictor 
variable and the estimated (predicted) score in the regression analysis (i.e., Yhat). Structure 
coefficients (in squared format) express how much variance in the predicted variable (Yhat) that 
the predictor variable possibly could explain. Because they are simple bivariate correlations, 
structure coefficients are not affected by collinearity as betas are (Thompson & Borrello, 1985). 

Because beta weights may not express accurate variable interpretation, structure 
coefficients are being supported as an important step in statistical interpretation (Courville & 
Thompson, 2000; Thompson , 2001). Cooley and Lohnes (Thompson, 1990) argue for the 
reporting of structure coefficients by the general researcher. Of course, structure coefficients 
yield the same interpretation as zero-order correlations between the predictor and the dependent 
variable (Knapp, 1978). However, structures are interpretation in terms of the synthetic Yhat 
variable. 

Method 

Multiple regression studies obtained from the Journal for Research in Mathematics 
Education and the School Science and Mathematics journal were identified. Every journal 
article of each issue was inspected for articles using multiple regression. I began with the latest 
issue and continued back until five complete volumes of each journal were exhausted. For the 
Journal of Research in Mathematics Education, I began with Volume 32, Number 4 and stopped 
after the inspection of Volume 27. For School Science and Mathematics , I began with Volume 
101, Number 7 and stopped after completing volume 96. After a multiple regression analysis 
was located, the article was read for the reporting and interpretation of effect sizes, Pearson 
correlation coefficients (r), and structure coefficients. This information was coded in a table. 
Also, the type of effect size was noted. 
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The first multiple regression from each article was selected for inspection. A study 
received credit for reporting the various information if the information was described in tables or 
in words. A study received credit for interpretation of effect sizes, Pearson correlation 
coefficient, and structure coefficients if an explicit description of or reference to the statistic was 
included. 

Results 

Nineteen multiple regression research articles were located and coded. Simple 
frequencies for observations are presented in Table 1 . 

Table 1. 



Summary of Reporting and Interpretation in Multiple Regression Articles. 





Report 






Interpret 




Effect 

Size 


Pearson Structure 

r Coefficients 


Effect 

Size 


Pearson 

r 


Structure 

Coefficients 


Yes 


16 (84%) 


8 (42%) 


1 (5%) 


16 (84%) 


9 (47%) 


1 (5%) 


No 


3 (16%) 


1 1 (58%) 


18 (95%) 


3 (16%) 


10 (53%) 


18 (95%) 


Total 


19 


19 


19 


19 


19 


19 



Approximately 84% of the articles reviewed reported effect size. The interpretation of 
the effect size was the same at approximately 84% of the studies. Every article that reported 
effect size, interpreted the effect size. Three studies did not report or interpret effect size. When 
the effect size was reported, R 2 was the statistic of choice in all cases. 

Pearson correlation coefficients (r) were reported in approximately 42% of the multiple 
regression articles. All articles that reported the Pearson correlation coefficient, also interpreted 
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the coefficient. One study interpreted Pearson correlation coefficients but did not report the 
statistic (Schwartz, 2000). 

The use of structure coefficients was as expected and almost non-existent. Only one 
multiple regression article reported and interpreted structure coefficients (Terry & Baird, 1997). 

Discussion 

These results of the investigation of multiple regression analyses in mathematics and 
science education journals disagree with other studies that suggest the reporting of effect size to 
be slowly occurring (Henson & Smith, 2000) and often neglected (DeVaney, 2001). When using 
multiple regression analyses, mathematics and science researchers appear to have been 
responsive to the recommendations of the APA Publication Manual (APA, 2001). However, as 
Kirk (1996) suggested effect reporting is likely higher for regression research due to the 
commonly known R 2 effect size routinely provided by statistical software outputs. 

In this review of the literature, the only effect size reported was R . This is consistent 
with a report by Fan (2001) that R 2 is the most popular effect size reported in several journals of 
psychology and Kirk’s (1996) review. Statistical software packages routinely present this 
information in their regression output, making R 2 readily available. Therefore, when regression 
is used, effect reporting tends to be more frequent. However, for analyses that statistical 
packages tend not to output effects, reporting is more sparse. This reality begs the question of 
whether effect reporting is a function of thoughtful researcher judgment or statistical output. 

Adjusted R 2 is also printed in the regression output of most statistical packages. 

However, adjusted R was not reported in the current study. Adjusted R is a “corrected” effect 
and will be less than or equal to the R . Corrected effects correct for sampling error and are 
better estimates of the true population effect. 
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Unfortunately, mathematics and science education researchers seldom examine structure 
coefficient to determine predictor importance. Zero-order correlations are also seldom 
interpreted beyond beta weights or unstandardized coefficients. Because statistical packages do 
not routinely report these coefficients in their outputs, structure coefficients are not as easily 
found (although they may be found by simply dividing the bivariate correlation between a 
predictor and the dependent variable by the multiple R). Because of this lack of reporting and 
interpretation, it is certainly possible that the researchers made erroneous conclusions about what 
predictors could have predicted the dependent variable. Courville and Thompson (2001) provide 
several examples of such misstatements. 

In sum, good research multiple regression practice should consider effect sizes when 
deciding if a result is noteworthy and both standardized weights and structure coefficients when 
deciding what predictors were valuable in generating the observed effect. The present review of 
two mathematics and science education journals suggest reasonable reporting of effects, with an 
equal amount of interpretation of those effects, and almost non-existent examination of structure 
coefficients. 
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